Scale-free memory model for multiagent reinforcement learning. Mean field approximation and rock-paper-scissors dynamics

A continuous time model for multiagent systems governed by reinforcement learning with scale-free memory is developed. The agents are assumed to act independently of one another in optimizing their choice of possible actions via trial-and-error search. To gain awareness about the action value the agents accumulate in their memory the rewards obtained from taking a specific action at each moment of time. The contribution of the rewards in the past to the agent current perception of action value is described by an integral operator with a power-law kernel. Finally a fractional differential equation governing the system dynamics is obtained. The agents are considered to interact with one another implicitly via the reward of one agent depending on the choice of the other agents. The pairwise interaction model is adopted to describe this effect. As a specific example of systems with non-transitive interactions, a two agent and three agent systems of the rock-paper-scissors type are analyzed in detail, including the stability analysis and numerical simulation. Scale-free memory is demonstrated to cause complex dynamics of the systems at hand. In particular, it is shown that there can be simultaneously two modes of the system instability undergoing subcritical and supercritical bifurcation, with the latter one exhibiting anomalous oscillations with the amplitude and period growing with time. Besides, the instability onset via this supercritical mode may be regarded as "altruism self-organization". For the three agent system the instability dynamics is found to be rather irregular and can be composed of alternate fragments of oscillations different in their properties.Comment: 17 pages, 7 figur

Stress-induced variation in evolution: from behavioural plasticity to genetic assimilation

Extreme environments are closely associated with phenotypic evolution, yet the mechanisms behind this relationship are poorly understood. Several themes and approaches in recent studies significantly further our understanding of the importance that stress-induced variation plays in evolution. First, stressful environments modify (and often reduce) the integration of neuroendocrinological, morphological and behavioural regulatory systems. Second, such reduced integration and subsequent accommodation of stress-induced variation by developmental systems enables organismal ‘memory’ of a stressful event as well as phenotypic and genetic assimilation of the response to a stressor. Third, in complex functional systems, a stress-induced increase in phenotypic and genetic variance is often directional, channelled by existing ontogenetic pathways. This accounts for similarity among individuals in stress-induced changes and thus significantly facilitates the rate of adaptive evolution. Fourth, accumulation of phenotypically neutral genetic variation might be a common property of locally adapted and complex organismal systems, and extreme environments facilitate the phenotypic expression of this variance. Finally, stress-induced effects and stress-resistance strategies often persist for several generations through maternal, ecological and cultural inheritance. These transgenerational effects, along with both the complexity of developmental systems and stressor recurrence, might facilitate genetic assimilation of stress-induced effects. Accumulation of phenotypically neutral genetic variance by developmental systems and phenotypic accommodation of stress-induced effects, together with the inheritance of stress-induced modifications, ensure the evolutionary persistence of stress–response strategies and provide a link between individual adaptability and evolutionary adaptation

Adaptive genetic complementarity in mate choice coexists with selection for elaborate sexual traits

Choice of genetically unrelated mates is widely documented, yet it is not known how self-referential mate choice can co-occur with commonly observed directional selection on sexual displays. Across 10 breeding seasons in a wild bird population, we found strong fitness benefits of matings between genetically unrelated partners and show that self-referential choice of genetically unrelated mates alternates with sexual selection on elaborate plumage. Seasonal cycles of diminishing variation in ornamentation, caused by early pairing of the most elaborated males, and influx of increasingly genetically unrelated available mates caused by female-biased dispersal, lead to temporal fluctuations in the target of mate choice and enabled coexistence of directional selection for ornament elaboration with adaptive pairing of genetically unrelated partners

Evolutionary significance of phenotypic accommodation in novel environments: an empirical test of the Baldwin effect

When faced with changing environments, organisms rapidly mount physiological and behavioural responses, accommodating new environmental inputs in their functioning. The ubiquity of this process contrasts with our ignorance of its evolutionary significance: whereas within-generation accommodation of novel external inputs has clear fitness consequences, current evolutionary theory cannot easily link functional importance and inheritance of novel accommodations. One hundred and twelve years ago, J. M. Baldwin, H. F. Osborn and C. L. Morgan proposed a process (later termed the Baldwin effect) by which non-heritable developmental accommodation of novel inputs, which makes an organism fit in its current environment, can become internalized in a lineage and affect the course of evolution. The defining features of this process are initial overproduction of random (with respect to fitness) developmental variation, followed by within-generation accommodation of a subset of this variation by developmental or functional systems (‘organic selection’), ensuring the organism's fit and survival. Subsequent natural selection sorts among resultant developmental variants, which, if recurrent and consistently favoured, can be inherited when existing genetic variance includes developmental components of individual modifications or when the ability to accommodate novel inputs is itself heritable. Here, I show that this process is consistent with the origin of novel adaptations during colonization of North America by the house finch. The induction of developmental variation by novel environments of this species's expanding range was followed by homeostatic channelling, phenotypic accommodation and directional cross-generational transfer of a subset of induced developmental outcomes favoured by natural selection. These results emphasize three principal points. First, contemporary novel adaptations result mostly from reorganization of existing structures that shape newly expressed variation, giving natural selection an appearance of a creative force. Second, evolutionary innovations and maintenance of adaptations are different processes. Third, both the Baldwin and parental effects are probably a transient state in an evolutionary cycle connecting initial phenotypic retention of adaptive changes and their eventual genetic determination and, thus, the origin of adaptation and evolutionary change

Data from: Signal architecture: temporal variability and individual consistency of multiple sexually selected signals

1. Multiple signals should be favoured when the benefit of additional signals outweigh their costs. Despite increased attention on multiple-signalling systems, few studies have focused on signal architecture to understand the potential information content of multiple signals. 2. To understand the patterns of signal plasticity and consistency over the lifetime of individuals we conducted a longitudinal study of multiple signals known to be under sexual selection in male lark buntings, Calamospiza melanocorys. 3. Within years, we found extensive among-individual variation in all four plumage ornaments we measured. Surprisingly, there were few correlations among these signals, suggesting that individuals contain a mosaic of signals. Signals were only moderately repeatable across years, which indicates some signal plasticity or age related change. In some years, the direction of change in particular signals relative to the previous year was consistent for most individuals in the population, suggesting that broad scale ecological factors affected the ornament phenotype. Different ornaments were affected by different ecological or social factors because the population-wide shift in a given signal was independent of change in other signals. 4. Our combined results suggest that different signals—including different components of the same color patch in some cases—provide diverse and independent information about the individual to signal receivers in the context of sexual selection